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Abstract — Direct evaluation of the rate-distortion function has 
rarely been achieved when it is strictly greater than its Shannon 
lower bound. In this paper, we consider the rate-distortion 
function for the distortion measure defined by an e-insensitive loss 
function. We first present the Shannon lower bound applicable 
to any source distribution with finite differential entropy. Then, 
focusing on the Laplacian and Gaussian sources, we prove 
that the rate-distortion functions of these sources are strictly 
greater than their Shannon lower bounds and obtain analytically 
evaluable upper bounds for the rate-distortion functions. Small 
distortion limit and numerical evaluation of the bounds suggest 
that the Shannon lower bound provides a good approximation 
to the rate-distortion function for the e-insensitive distortion 
measure. 

I. Introduction 

In source coding, the rate-distortion function R(D) of 
a source shows the minimum information rate required to 
reconstruct the source outputs with average distortion not 
exceeding D. Rate-distortion functions have been explicitly 
evaluated for various sources and distortion measures. The 
Shannon lower bound (SLB) Rl (D) plays an important role in 
the explicit evaluation of rate-distortion functions of difference 
distortion measures. A common approach is to derive Rl(D) 
and examine the condition for R(D) to coincide with Rl(D) 
[4|, |1|. There have been, however, only several results when 
R(D) > Rl(D) for all D. In this case, direct explicit evalu- 
ation of R(D) has been achieved only in limited cases such 
as discrete memoryless finite-alphabet sources JT] and a class 
of sources under an absolute-magnitude distortion measure 
0, lfl2l . ifTTI . There have also been indirect approaches. 
Rose proposed a deterministic annealing algorithm to generate 
R(D) based on the fact that under the squared distortion 
measure, the optimal reconstruction is purely discrete when 
R(D) > Rl(D). Buzo et al. obtained upper and lower bounds 
for R(D) under the Itakura-Saito distortion measure |]2]. 

In this paper, we focus on the e-insensitive loss function as 
a distortion measure, which was introduced to support vector 
machines for regression ifTOl . We obtain the SLB for this 
difference distortion measure, which is analytically evaluable 
for arbitrary sources with finite differential entropy. Then, 
we examine the condition for the rate-distortion function to 
coincide with the SLB. Taking the Laplacian and Gaussian 
sources as specific examples, we prove that the rate-distortion 
functions of these sources lie strictly above their SLBs for 
all D when e > and derive analytically evaluable upper 



bounds for the rate-distortion functions. Investigation of small 
distortion limit of these upper bounds shows that the SLB has 
the accuracy of 0(e 2 ) as D — > in both sources. Numerical 
evaluation of the lower and upper bounds demonstrates that 
the SLB gives a good approximation to R(D) for small 
distortion while the trivial upper bound provided by the rate- 
distortion function for e = suggests that the SLB also gives 
a reasonable approximation for large distortion as well. 

II. Rate-Distortion Function for the ^Insensitive 
Distortion Measure 

A. Rate-Distortion Function 

Let X and Y be random variables on R and d(x, y) be the 
non-negative distortion measure between x and y. The rate- 
distortion function R(D) of the source X ~ p(x) with respect 
to the distortion d is defined by 



R(D) = inf I(q), 

q{y\x):E[d{x,y)\<D 



(1) 



where 

m = 



q(y\x)p(x) log 



-dxdy 



J q(y\x)p(x)dx 

is the mutual information and E denotes the expectation with 
respect to q(y\x)p(x) . R(D) shows the minimum achievable 
rate for the i.i.d. source with the density p(x) under the given 
distortion measure d fl], ||4). 

The above minimization problem can be reformulated as a 
problem of minimization over the reproduction density q(y) 

CD, 0, 



inf 

i{y) 



p{x)\og I exp(sd(x,y))q(y)dydx 



(2) 



where s < is a parameter. Then, if there exists q s (y) that 
achieves the infimum in Eq. (0, R(D) is parametric ally given 

by 



R(D S ) 



p(x)\og / exp(sd(x,y))q s (y)dydx + sD s , 



(3) 



p(x)q s (y\x)d(x,y)dxdy, 



where the optimal conditional density of reconstruction, 
q s (y\x) is defined by 

q s (y) exp(sd(x,y)) 



Qs(y\x) 



J q s (y)exp(sd(x,y))dy' 



(4) 



In Eq. ([3), R(D) is parameterized by s < 0, which corre- 
sponds to the slope of the tangent of R(D) at (D s , R(D S )) 
and hence is referred to as the slope parameter [fl]. 

From the properties of the rate-distortion function R(D), 
we know that R{D) > for < D < L> max , where 



-Dm 



= inf 



p(x)d(x, y)dx, 

CD p. 90]. 



(5) 



and R(D) = for D > D max 

B. e-Insensitive Loss Function 

In this paper, we focus on the following difference distortion 
measure defined by the e-insensitive loss function p e (Fig. [T), 



where 



d(x,y) = p £ (x-y) 

Pe{z) = 



(6) 



\z\-e, 
0, 



(\z\>e), 
(\z\<e). 



Pe(z) 



-e e z 

Fig. 1. The e-insensitive loss function p e (z). 

This loss function with e > was introduced to support 
vector regression in order to provide a sparsity inducing 
mechanism iflOl . (8). We denote the rate-distortion function 
for this distortion measure by R^(D) and the maximum 
distortion D max in Eq. (0 by 



y max- 



C. Shannon Lower Bound 

Generally for difference distortion measures, Shannon ob- 
tained a lower bound to R(D), which is referred to as the 
Shannon lower bound (SLB) [1, p. 92]. For the e-insensitive 
distortion measure, it is parametrically expressed as 

h(p) - h(g s ), (7) 

D s = I p £ (x)g s (x)dx, (8) 



R (£ \D S )>R<£\D S ) 



where h(p) = — J p(x)\ogp(x)dx is the differential entropy 
of the probability density p and g s is the probability density 
function defined b)Q 

9s{x) = Je^dz - (9) 

We explicitly evaluate h(g s ) to obtain the SLB. The density 
g s is explicitly given by 

(x < — e), 

!i,U) = { r-- (M <e), (10) 

{x > e), 




We omit the dependency on e in notations unless we put e = 0. 



where 



C s 



.1+ s 



Its differential entropy is evaluated as, 

1 + Isle 



h(g s ) = log 2 



1+ se 



(11) 



(12) 



The slope parameter s is related to the average distortion D s 
by Eq. dHJ, which is rewritten as, 

2 1 

^ = = (l + e\ S \M (13) 

Solving this for \s\ yields 



-As + y/D 2 a + AD s e 



2D s e 

Putting this back into Eq. (fT2l . we have 



h(g s ) = \og{2e + D s + y/Dl+AD s e)- 



D s - ^Df+Il\e 



2e 

Thus, from Eq. O, we obtain the following theorem. 

Theorem 1 The rate-distortion function for the e-insensitive 
distortion measure in Eq. © satisfies R {e \D) > R { [\D) for 
all D, where 



R ( l\D) = h(p) - log(2e) - log ( 1 + D + Vf) 2 + 2D 



+D - V£> 2 + 2D, 

D = ^ and h(p) is the differential entropy of the source 
density. 

D. Condition for R^(D) = R { [ ] \D) 

For any negative value of the slope parameter s, the lower 
bound Rf/(D 8 ) coincides with R^ e \D s ) if and only if the 
condition 

(14) 



P{x) = / l(v)9s(x - y)dy, 



holds for all x and a valid density function q(y) |Q] p. 94]. 
The condition in Eq. ( TBI is equivalent to 



P(w) = Q(u)G,(w), 



(15) 



where P, Q and G s are the Fourier transforms (characteristic 
functions) of p, q and g s respectively. 

The Fourier transform of g s in Eq. dTOb is specifically given 

by 



G.(w) = 



,sp e (x) 

-cT' 



I i sin(u;e) . / \ 



1 + 8 



= L\.\(u)-M${u). (16) 
Here, the first factor, defined as L\ s \ (uS) = S 2 S +LJ 2 , is the char- 



acteristic function of the Laplace distribution with parameter 
\s\ whose density function is l\ 8 \(x) — ^-e s \ x \. The second 



factor, M$(u) 

function of the mixture of the delta distributions (on — e and 
e) and the uniform distribution on [— e, e] mixed with the 
proportion 1 : e\s\. More specifically, the density function of 
this mixture is expressed as 

( e) 1 S(x - e) + 6(x + e) 

m w (l) = T+iffi 2 



e\s\ 



l + \s\e 



-t-cos(uje) 



, is the characteristic 



e s 



1 + ss 



where 6 is the Dirac delta function and itr_ £iE i is the density 
function of the uniform distribution on [— e, e]. Hence, Eq. ( [ToT l 
means that the density g s is given by the convolution / 



(<0 



of Z | s | and 



rn 



00 
1*1 



M* TO |.| 

Summarizing Eqs. (Q3) and ( fTSI l. we see 



that for the e-insensitive distortion measure, the condition for 



(D) to coincide with (D) is the existence of a valid 
characteristic function Q{uS) satisfying 

P(cj) = Q(cj)L 1s1 (uj)M$(lj), 



(17) 



for the characteristic function P(ui) of the source distribution. 

The above condition is rewritten as P(lS)/L\ 8 \(uS) = 
M^) (uj)Q(uj). If Q{lo) is the characteristic function of a 
probability distribution, that is, the Fourier transform of a 
density q(y), then the right hand side is that of convolution 
of q and mfj . Hence, Eq. ( fTTT i is never satisfied for any 
density q(y) unless P(uS)/L\ 8 \ (lj) is the Fourier transform 
of a valid density. This means that a necessary condition for 
R&(D S ) = R { [\D S ) is given by R m {D s ) = R { °\d s ), that 
is, the SLB coincides with the rate-distortion function under 
the absolute-magnitude distortion (e = 0). 

We will use the above condition in Section [HI] to prove 
Rj/(D) is strictly smaller than R( E \D) for the Laplacian 
and Gaussian sources. 

E. General Upper Bound 

Let us turn to upper bounds for R( e \D). Since p e {x) < 
Po(x) — \x\, we have a trivial upper bound, 

R^(D) < R(°\D), 

where R( \D) is the rate-distortion function for the absolute- 
magnitude distortion measure, d(x, y) = \x — y\. 

Another more informative upper bound is obtained by taking 
q(y\x) — g s (y — x), where g s is defined by Eq. (O, in the 
original rate-distortion problem in Eq. ([T] [1. p. 103]. This 
yields the following upper bound, 



R [£) (D S ) < R ( u\D s ) = h{r s ) - h(g s ), 



where 



r a (y) = (9s * p)(y) = 9s{y- x)p(x)dx 



(18) 



(19) 



and D s is given by Eq. (O and further by Eq. (fTTt . Note in 
Eq. ( fT~8b that the term h(g s ) is common to the SLB and is 
specifically given by Eq. (fT2l) . 

Since g s is defined by p e as in Eq. (O, h(r s ) is analytically 
intractable for many sources. Hence, we create a further upper 
bound which is analytically obtained for any souces with finite 



variance. 



The Gaussian distribution with variance v maximizes dif- 
ferential entropy among the distributions whose variance is 
constrained to be v. The maximum value of differential 
entropy is | log(27rew). Therefore, the differential entropy 
h{r s ) = h(g s * p) is upper bounded as follows, 



h(r s ) < ilog(27re(u p + i;£ £ >)) , 

where v p = J x 2 p(x)dx — (J xp(x)dx) 2 and vS;^ 
/ x 2 g s (x)dx. This is because the variance of g s *p is v p 
The variance of g s is specifically evaluated as, 



(20) 



C sl 3 > ^ ' (21) 

where C s is defined in Eq. (JTTJ. The general upper bound 
in Eq. ( fTSl . combined with Eqs. (l20l and (|2T1 i. yields the 
following upper bound to R { u(D), 

Rge(Ds) = \ log (2Tre(v p + v^)) - h(g s ), (22) 

which is referred to as the Gaussian entropy bound. 

In the next sections, we will evaluate these upper bounds for 
the Laplacian and Gaussian sources to examine the tightness 
of the general lower bound obtained in Theorem Q] 

III. Laplacian and Gaussian Sources 

A. Laplacian Source 

In this subsection, we consider the Laplacian source with 
parameter a, 



p(x) = l a (x) 



(23) 



The SLB for this source is given by Theorem Q] with the 
differential entropy, 

hip) = 1 - log | . 
The maximum distortion in Eq. (0 is 



max 



p e (x)p(x)dx — — e 
a 



(24) 



For the absolute-magnitude distortion measure (e = 0), 

R {0) (D) = R { °\D) = -log(aD), (0<D<l/a), (25) 
holds [ 1 , p. 95, Example 4.3.2.1] because the condition in 



Eq. (H7) reduces to M^)( 



Q(u) = 



1 and 

„2 



1 - 



a 



which is the Fourier transform of the valid probability density, 

q(y) = f£<%) + la{ - y) - For e > °' however ' 

R( 6 \D) is strictly greater than R[ (D) for all D, which we 
will prove in the following. We will later derive an analytic 
upper bound to R^ (D) from Eq. ( fT8l . 

The condition for R^1(D) 
equivalent to 



R { l\d) in Eq. dTTJ) is 



Q(w) = 



l \s\ 



1 - 



For |*| > a, {^i + (l - ^i) > ^i. Putting W = 

(2fc — 1/2)—, where fc is a natural number, we have 



Q((2fc-l/2)- 



> 



a 2 1 +e|s| 



e s 



(2jfe-l/2)7T. 



< 0, 



Eq. (HI that R^'(D s ^ a ) = 1 - log(l + as) - j 

which implies that R( E \D) is strictly greater than R^(D) 
for all D. 

To obtain an analytic upper bound for R( E \D), we evaluate 
h(r s ) in Eq. il Si . The density r s (y) defined in Eq. ( fT9] > is 
specifically given by 

[ 2Bj & s(-y), (f ^ ~ £ )' 

where a s (y) = ^e^^ + ^e a ^ + 2 and b s (y) = 

_2_ e -ct(y+s) + s e -a(y-e) + _2a^ s(y-e) gj th djf . 
a — s a+s q^-s^ 

ferential entropy of r s (y) is not analytically simplified any 
more, we evaluate it from above to obtain an upper bound for 
R^(D S ). 

First, we bound a s (y) and b s (y) from below. 



a s (y) > 2 



2se" 



a — s 

for \y\ < e and since s > —a, 



■ cosh(ae) = c s 



Uv) > 



,-a(y+e) _|_ 



-a(y-e 



2a 2 



,-a{y-e) 



a — S 

c s e a{e - y} 



a + s 



for y > e. Next, let us define 



b s (y)dy 



-lae 



a — s a 



s 2 (a - s) - 2a 3 
(a 2 — s 2 )sa 



Finally, h(r s ) is bounded as follows, 

h(r s )<-log^-^B s 



cl Es 



(26) 



where 



yb s {y)dy 



S 1 + a£ c -2ae 



1 + ae 



2a 2 



1 — se 



which easily follows from J°° ye ay dy = 1+ " £ e a£ . 



Thus, we have the further upper bound, R$'(D a ) < 



R%(D S ), where 



= -log 



c s ae 

2Ce C i 



a 



■E, - h{g s ), (27) 



which we will refer to as the analytic upper bound in Section 
E3 



Cs 



In the low distortion limit, D — > and \s\ 

• 2s, c s -> 2e~ Q£ sinh(ae), B s -+ \ 
i±f^2e- Qe sinh(ae). Then, Eq. 



h(r-oo) <ae- log 



sinh(ae) e 



Since the right hand side becomes arbitrarily large for k —> oo, 
Q(oS) can not be a Fourier transform of any density q(y). This 
means that R { [ ] (D s ) < R^ (D s ) for s < -a. It follows from e" 



2c 



of 



Furthermore, it follows from Slnh ( a£ ) = \ 

ae 



oo, we have 
f e _ae sinh(ae), 
turns out to be 

■ sinh(ae). 

0(e 4 ) and 



(ae 



= 1 — ae - 



(ae) 



fc(r_oo) < + 



0(e 3 ) that 



+ 0(e 3 ), 



which implies that the SLB R ( [ ] '(£>) satisfies i?^ } (0) < 

R {E) {0) < Rl ] {0) + ^f- + 0(e 3 ). 

B. Gaussian Source 

In this subsection, we consider the Gaussian source with 
mean zero and variance a 2 , 



p(x) 



1 



2ncr 2 



(28) 



The differential entropy of this source is h(p) — i{l 
log(27rcr 2 )}. The maximum distortion in Eq. (|5j is 



D (e) 



p £ (x)p(x)dx = 2 {a 2 p{e) - e<Z> c (e/a)} , (29) 



where $ c (x) = e~ t2 / 2 dt. We show that R^{D) 

for this source lies above its SLB for all D and consider 
small distortion limit of the Gaussian entropy upper bound 
in Eq. (l22l >. 



The condition for R {e) {D s ) = R ( ^{D S ) is given 
by Eq. flTJ with P(u) = cr* 2 " I 2 . The inverse 
transform of P[w)/L\ s \(w) = (l + -^jP{u) is 

p(x) ( 1 + \ „ \ l„a ~ iJfc„4 ) » which becomes negative for 



large s|j From the discussion below Eq. (fTTI i. this implies 
that R^(D) > R { ° } {D) and hence R^(D) > R^\d) for 
all D. 

We have the Gaussian entropy bound in Eq. d22l > with w p = 
(j 2 . In the limit, |s| — >• oo, ui E — >• e 2 /3. Then, the upper bound 
in Eq. (l20i i is further upper bounded as follows, 



2?re 



since log(l + x) < x for x > 0. This means that the SLB 
given by Theorem Q] provides an approximation to R^ (D) 
with accuracy as 13 —> 0. 

IV. Numerical Evaluation 

Figure |2]depicts the functions (D) and R%(D) for the 
Laplacian source in Eq. (l23l l with a = \/2 when e = 0.1. It 
also shows Rq E {D) in Eq. (l22l i with v p = 2/ a 2 = 1 and the 
trivial upper bound R^(D) given by Eq. (|25| ). It is observed 
that the analytic upper bound and the SLB are very close to 
each other for small distortion (D < 0.01). The analytic upper 

2 See also Eq. (4. 3. 27) in fl. p. 97]. 
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Fig. 2. Rate-distortion bounds for the Laplacian source. 
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Fig. 3. Rate-distortion bounds for the Gaussian source. 



bound becomes looser than the Gaussian entropy bound for 
large distortion (0.05 < D) while the trivial upper bound is 
relatively more informative about R^(D) in the vicinity of 
Dml x when combined with the SLB. This suggests that the 
SLB provides reasonable approximation to R^(D) even for 



large distortion. Let -D ma x denote the average distortion where 
the SLB R ( f\D) reaches zero. We observed the following 
values of Z)£ a x L , £>max defined in Eq. (l24l > and Anax = l/a, 
= 0.6136 < D&L = 0.6139 < D^L = 0.7071. 
Figure H presents the bounds R ( [ ] (D) and R^ E (D) (e = 
0.1) and R^(D) for the Gaussian source in Eq. (128b with 
<j 2 = 1. The rate-distortion function R^(D) for the Gaussian 
source was evaluated according to its explicit parametric form 
given in |9] Th. 2]. It can be seen that the Gaussian entropy 
bound is very tight for small distortion (D < 0.1), which 
implies high accuracy of the SLB and that the trivial upper 
bound is informative about R^(D) around D = £>max- The 



observed values of -Dmlx", fmL defined in Eq. (l29T l and 
— ^2a 2 /n are as follows, Dmax" 
0.7979. 



D 



0.6662 < D^L 



V. Conclusion 

In this article, we have shown upper and lower bounds for 
the rate-distortion function of the e-insensitive distortion mea- 
sure. We derived the SLB, which is applicable to any source 
densities. Focusing on the Laplacian and Gaussian sources, we 
have proved that the rate-distortion functions for these sources 
are strictly greater than the corresponding SLBs for all D and 
provided upper bounds for the rate-distortion functions, which 
are proved to have accuracy of 0(e 2 ) in the small distortion 
limit. We have demonstrated through numerical evaluation that 
the SLB is very accurate in the small distortion region while 
it still provides reasonable approximation to R( e '(D) for the 
high distortion region around £>max as R( \D) suggests. 

In order to explicitly evaluate R^(D), it would be im- 
portant to explore properties of the optimal reproduction 
distribution q s (y)- It is also important to develop practical 
learning algorithm for the mixture J g s (x — y)q(y)dy, where 
g s is defined by the e-insensitive loss function as in Eq. ( TTOb . 
Another issue to be addressed is the extension of our results 
to vector sources by using an extension of the e-insensitive 
loss function to higher-dimensional vectors. There are some 
variations of the e-insensitive loss function Q, 0. It would 
also be an interesting undertaking to investigate properties of 
the rate-distortion functions for these variations. 
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